Data Extraction
Sensor data in SCS is collected by the primary acquisition service ACQ. ACQ then writes the data out to a number of different targets for ingestion/use by other external parties or subsystems inside SCS itself. These targets are subject to change as SCS evolves, but at the time of this writing they are as follows:
- SignalR
- Database
- Files
- UDP
- NetCDF
Note: NetCDF has been temporarily removed from SCS pending formatting agreements between OMAO and NCEI. Though present in this documentation it will not be available for use in SCS until the agreements are in place.
SignalR
SignalR is an open-source library that simplifies adding real-time web functionality to apps. Real-time web functionality enables server-side code to push content to clients instantly. It provides an API for creating server-to-client remote procedure calls (RPC). The RPCs call JavaScript functions on clients from server-side .NET code. More information can be found in the SignalR section of the help.
Database
The backend of SCS is a MySQL database and is the backbone of the entire system. In regards to this section the database is the home of all recorded data ranging from sensor data collected by ACQ and event data collected by the Event Logger to various system logs and beyond. The database is meant to be non-interactive from a user perspective and is simply an internal store for SCS itself. For instance, to get data from the database SCS provides various tools for your use rather then requiring you to run SQL queries yourself. Sensor data stored in the database can be extracted via the Extraction / To Disk page located in the Data Management menu. Instructions to do this are below in the NetCDF section.
Files
In the past SCS has written data directly to disk in the form of RAW files. The current version of SCS does this as well for backwards compatibility as many other 3rd party apps have been written around these files and time will be needed for them to be updated. Long term the target input for these apps should migrate over to NetCDF however as RAW files will be rendered obsolete at some point in the future.
RAW files are direct dumps of the sensor feeds to a file. These files are located in optional sub-folders as defined by the user via CFE (default root is D:\SCS\DATA\RAW DATA). Each file corresponds to a single days worth of data from a single message. As it's a direct write of the sensor feed it's impossible to tell what the format of the data is, however in the majority of cases it is NMEA compliant and could be considered a comma-delimited data-set.
The latest SCS adds a header line to each RAW file that was not present in prior versions of SCS. Please modify any ingestion routines to accommodate!
In the event you wish to modify the timestamp used when generating the file names you can do so by manually modifying the .config file for the ACQ service. Inside the file there is a setting named OutputFileNameFormat whose default value is '3'.
By modifying the value (and restarting ACQ) you will set the naming convention used for the RAW files as detailed below. Note this does not effect the timestamp of the RAW data itself which is locked into the ISO 8601 format.
| Value |
Example Result |
| 0 | 2019-02-08 19.25.24Z SIMRAD MX512 GPS-MX512 #2 GPS-$GPGGA.RAW.log |
| 1 | SIMRAD MX512 GPS-MX512 #2 GPS-$GPGGA 2019-02-08 19.25.24Z.RAW.log |
| 2 | 1550843658_SIMRAD_MX512_GPS-MX512_#2_GPS-$GPGGA.RAW.log |
| 3 | 20190208T192524Z_SIMRAD_MX512_GPS-MX512_#2_GPS-$GPGGA.RAW.log |
UDP
ACQ automatically sends data out via UDP Broadcast if an admin has requested it to do so. Details on this are described in the UDP section.
NetCDF
See note above regarding temporary removal of NetCDF functionality from SCS.
NetCDF is the format most data pulled from SCS should be in. It has become the format requested by NCEI and many scientists and so SCS has switched from RAW files to NetCDF as the primary output. SCS writes sensor data to NetCDF both in real time and as requested by users. SCS complies with the templates set forth by NCEI ( https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/ ) however as we do not wish to interpolate data we cannot use any feature type other than a point. Unfortunately this may not be considered very useful by end users. Future versions of SCS may add an interpolated export and/or NCEI themselves may run routines to take the RAW SCS NetCDF and generate a more useful output.
There are many tools out there for opening and viewing NetCDF files. Please work with your system administrator to see which one you should use. If you are an OMAO user the one on the approved software list is Panoply - https://www.giss.nasa.gov/tools/panoply/
As ACQ writes data to the above outputs SCS also starts building out a NetCDF version of the data in real time. While this is redundant it does save time post-cruise when an entire cruise worth of data is needed since the dumps will have already been created. However, in the event the real-time files fail or have errors SCS also allows manual building and extraction requests to be issued to pull the data directly from the database and dump it to disk in NetCDF format.
To download daily sensor data NetCDF files open the Extraction / To Disk page located in the Data Management menu. A list of completed NetCDF files will be displayed. If you want to pull one simply click the Download button at the end of it's row.
If the date you want is not present in the list (and is not pending on the list on the right) you can submit a new job to extract and build NetCDF files by entering a start/end date range and clicking the Submit button.
If you are a server admin and have access to the filesystem the NetCDF files are stored by default in the D:\SCS\DATA\NetCdf folder.
NetCDF files that are in the process of being built will be displayed in the 'Pending' grid on the right side of the screen. Once they've completed the files will become available for download.
Much like hitting a button in an elevator, repeatedly hammering it / adding the same day over and over will not actually speed up the processing job so please be patient.